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ABSTRACT 



The Illinois State Board of Education partnered with two 



outside groups, the National Computational Science Alliance (NCSA) and the 
North Central Regional Technology in Education Center (NCRTEC) , to provide 
staff development, along with technological software, hardware, and support, 
to improve 6th graders' performance in the areas of mathematics, science, and 
reading. The project involved teachers, administrators, and support staff 
from diverse public and private schools that contained students of low 
socioeconomic status, had high transfer rates, and large percentages of 
students receiving special education. Participants received 5 days of 
training in the engaged model of learning and in the use of specific modeling 
software. Participant communication was enhanced through e-mail and other 
web-based technologies. Teachers planned, implemented, and evaluated action 
research projects at their schools that involved students in collecting data 
and highlighted a real problem. Performance assessment measures were utilized 
to gauge project merit. Feedback from participants was used to improve future 
efforts at local school sites in achieving state and local standards related 
to data collection, representation, and interpretation. Evaluation activities 
produced confirming evidence for each of the three major goals (helping 
students achieve high academic standards, providing staff development 
opportunities for teachers, and establishing community collaboration among 
participating 6th graders) . An executive summary is appended. (Contains 13 
references.) (SM) 
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Project Overview 

The Illinois State Board of Education has partnered with the National 
Computational Science Alliance (NCSA) and the North Central Regional 
Technology in Education Center (NCRTEC) to provide staff development training 
along with technological software, hardware and support to improve the 
performance of over 1000 sixth grade students from across the state in the areas of 
mathematics, science and reading. The project involved teachers, administrators, 
and support staff from eight public school districts and three private schools from 
thirty-eight different classrooms and eight distinct regional communities. 

The ISBE provided about $250,000.00 to fund the project in FY99 through a 
competitive Technology Literacy Challenge grant request for proposal process. The 
grant monies provided staff development training, on-site support and actual 
technological software and hardware for teacher and student use. 

The classrooms selected for the project contained students reflecting the attributes of 
low socioeconomic status, high mobility rates in transferring both in and out of 
school, and a large percentage of students receiving special education services. The 
program participants were provided five days of training in the engaged model of 
learning and in the use of specific modeling software (i.e. both Stella and Model-It). 
Communication among participants was enhanced through the use of e-mail and 
other web-based technologies. 

Teachers were asked to plan, implement and evaluate an action research project at 
their school involving students in collecting data and information on a real world 
problem of choice. Projects completed included: energy audits in which students 
devised ways to decrease the cost and consumption of electrical energy; a recycling 
project designed to manage student use of paper and related implications for saving 
trees and reducing local landfill waste; and, a collaboration with a local park district 
to monitor and improve the water quality of a recreational lake. 

Performance assessment measures were utilized to gauge the merit and worth of the 
project. Feedback from the program participants was utilized to improve future 
efforts at local school sites in improving student performance in achieving state and 
local standards related to data collection, representation and interpretation. 



Participants 

The AHAs project involved 8 public school districts and 3 private schools from 
around the state. 38 sixth grade participated from the various schools. The schools 
were selected based upon a matrix of student attributes that included low SES, 
mobility rate, along with bilingual and minority representation. These 
characteristics were selected to address the ability of varying and unique student 
populations to achieve the Illinois Learning Standards through tire use of 
technology. 

Listed below are the districts and school participants, followed by a chart displaying 
just some of the population attribute characteristics. 
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Aurora East School District #131 

• Our Lady of Good Counsel School 

Belleville School District #118 

• Union School 

Carpentersville School District #300 

• Carpentersville Middle School 

Cahokia School District # 187 

• Centerville Elementary School 

Crystal Lake School District #47 

• Hannah Beardsley Middle School 

• Lundahl Middle School 

• North Middle School 

Mt. Vernon School District #80 

• Dr. Andy Hall School 

Palatine School District #15 

• St. Colette School 

• St. Thomas School 

• St. Theresa School 

• Virginia Lake School 

Wabash School District #348 

• North Middle School 



Student Population Attribute 


Data 


District 


School 


Hispanic 


Black 


Low 

Income 


LEP 


Belleville 


Union 


0.5% 


27.5% 


43.1% 


0.0% 


Carpentersville 


Middle 


37.8% 


10.7% 


34.2% 


26.1% 


Cahokia 


E. Morris 


0.7% 


57.6% 


73.3% 


0.0% 


Cahokia 


Centerville 


0.0% 


87.1% 


73.0% 


0.0% 


Crystal Lake 


North 


3.8% 


0.1% 


9.3% 


0.0% 


Crystal Lake 


Lundahl 


2.0% 


0.9% 


3.3% 


0.0% 


Crystal Lake 


Beardsley 


5.8% 


0.7% 


8.3% 


1.2% 


Mt. Vernon 


Andy Hall 


2.2% 


26.8% 


48.0% 


0.0% 


Palatine 


Virginia Lake 


30.5% 


10.3% 


32.4% 


13.2% 


Wabash 


North 


0.0% 


1.0% 


33.6% 


0.0% 
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Project Goals 

The partnership of ISBE, NCSA and NCRTEC sought to create a technology rich 
environment in each of the participating schools to foster and facilitate student 
attainment of high standards in mathematics, science and reading. To this end, the 
partnership committed itself to the continual and on-going staff development of 
teachers in the latest technologies, and to the creation of model curricular materials 
for teacher and student use and implementation. 

There were three main goals of the project. They focused on helping teachers to use 
both engaged learning strategies and modeling software with students, and to 
provide action research opportunities to aid in school improvement planning. 

Other goals of the project dealt with the development and dissemination of 
instructional materials for use with other students across the state, facilitating 
communication about curriculum planning through web-based technologies, and 
the development of an accountability system to measure the extent to which student 
indicators of success and program activities resulted in attainment of stated 
instructional goals. 

The three stated AHAs project goals and some of the major project activities related 
to each of these goals are stated below: 

Goal 1 : Help students achieve high academic standards in math, science and 
reading. 

Project Activities 

• form a leadership team of representatives from each of the participating 
schools to serve as a steering committee for the project, 

• create a school leadership team to oversee local project activities, 

• agree upon local student indicators of success for the project, 

• provide teachers with training on engaged learning, and 

• have teachers plan and implement an engaged learning and technology 
rich learning project. 

Goal 2 : Provide staff development opportunities for teachers to develop, 
implement and evaluate technology-rich reality-based learning projects. 

Project Activities 

• provide background information, group training, and individual support 
for teachers to gain expertise in engaged learning, 

• provide training for teachers in utilizing internet telecommunication, 

• provide teacher training in action research and performance-based 
assessment measures, and 

• have teachers and staff share findings through collaborative planning. 

Goal 3 : Establish community collaboration between participating 6th grade 
students. 
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Project Activities 

• provide training for teachers in utilizing internet telecommunication, 

• enable teachers to share via websites and e-mail, and 

• model ways for teachers to utilize these internet telecommunication 
strategies as part of the classroom curricula. 



Engaged Learning 

Each of the staff development sessions provided teachers with information and 
opportunities to investigate and experience the instructional process known as 
engaged learning. This pedagogical strategy has been found to increase the active 
engagement of students during instruction. For this reason, it was chosen as a way 
for teachers to help students reach high standards of instruction. The engaged 
learning models have strong research support for the improvement of teaching and 
learning (Conrad 1996; Roth, 1994; Shuell, 1990). 

The engaged learning model enables teachers to address student outcomes, define 
activities to engage the learner; select instructional strategies and determine 
performance assessment measures in a systematic planning process to ensure 
student learning. 

Teachers were asked to develop and implement an engaged learning unit as the 
culminating activity of this project. All of the teachers successfully utilized the 
engaged learning model with their students of varying backgrounds and abilities. 
The Illinois learning Standards were utilized as benchmarks to gauge successful 
implementation of the engaged learning units. 



Action Research 

The grant provided teachers participating in the project with a day's training in 
action research. The action research model allows for classroom teachers to collect 
information about specific problems of teaching and learning, evaluate the 
information gathered, and then utilize the information to improve their instructional 
practice. 

Action research differs from other forms of educational inquiry in that the researcher 
is also the classroom practitioner. Thus through action research teachers act as 
researchers in the classroom, gathering evidence and making judgments in an 
attempt to reflectively improve a practice of teaching and learning (Sagor, 1992; 
Schon, 1983). 

The action research model often uses a cycle of phases to ensure proper 
implementation. These phases are: 1) defining a problem of choice, 2) gathering 
information about the problem, 3) planning a strategy of intervention, 4) collecting 
classroom information and evidence about the intervention, 5) making judgments 
and inferences about the data collected, and, 6) refining the practice based upon the 
evidence. 
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Performance-based Assessment 

Criterion-referenced testing (CRT) has been popular with educators for many years 
since it allows the alignment of assessment measures with instructional goals. In 
this way teachers and school districts have the ability to define and articulate a 
successful level of student performance to indicate mastery or attainment of a given 
outcome or objective. Whenever a teacher assesses what has been taught in the 
classroom, criterion-referenced testing is being used. 

CRT's blend nicely with projects of school improvement and curricular alignment at 
the school and district level for they continue to allow teachers to select indicators of 
success that are real and meaningful to student learning. In the past decade, a new 
form of criterion-referenced testing ( i.e. performance-based assessment) has become 
an acceptable standard of practice within the educational community. 

Performance-based assessment often allow students to demonstrate what they know 
about a particular topic or subject and to show how it can be used or applied. This 
concept of what we want students "to know and do" is at the heart of standards- 
based assessment. Besides knowledge and performance, students can also portray 
their beliefs and attitudes about a subject also know as disposition. 

The AHA project facilitated the development of performance-based measures to 
assess student learning and provided practice in these procedures through 
implementation of both the engaged learning and action research training modules. 
Various methods for collecting student performance data are always available for 
teacher use. Project participants were encouraged to utilize a combination of two or 
more of these data collection methods at any given time. Multiple assessments of 
student performance provided further evidence of student achievement and 
strengthened the performance-based assessment procedure. Some broad categories 
and specific kinds of acceptable performance-based indicators of student success are 
listed below (Rudy, 1999): 

TFACHER OBSERVATION 

(e.g. protocol rating sheet; checklist; videotaped lesson); actual observation of 
students by teacher during the activity related to previously established 
criteria 

TEACHER REFLECTION 

(e.g. teacher judgement; anecdotal records; journal); judgments and 
comments made about a student' s performance and / or level of 
understanding after an activity/ episode has been completed based upon 
previously stated criteria 

TEACHER ARTIFACT 

(e.g. course syllabus; problem-based learning activity; home web page; 
attendance record); item of record produced by the instructor utilized during 
the teaching and/ or learning activity/ episode based upon predetermined 
criteria for student performance 
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STUDENT REFLECTION 

(e.g. student journal; structured interview; survey); self-assessment and 
appraisal of performance and/or level of understanding (prior student 
training, practice and discussion of assessment criteria is necessary to use this 
procedure) 

STUDENT ARTIFACT 

(e.g. contents of a portfolio; lesson plan; essay; criterion-referenced test; group 
project); assessment completed based upon review and analysis of student's 
actual work; items produced through participation in specific learning 
activities; assessment criteria must be established prior to evaluation of 
materials; portfolio must be available for future reference and data 
verification (i.e. may be stored on CD-ROM, computer diskette, etc.) 

PEER REVIEW 

(e.g. observational checklist; performance rating sheet; feedback from an 
observer); peer assessment/ appraisal of another student's performance 
and / or level of understanding (prior student training, practice and 
discussion of assessment criteria is necessary to use this procedure) 



Modeling Software 

An integral part of the project involved the training of program participants in the 
use of modeling software. The AHAs project provided teachers of sixth graders 
with two days of training at the NCS site in Excel spreadsheets, Model-It software 
and Stella software. These software applications were utilized for their capabilities 
in aiding in data collection and interpretation, along with the distinct use of the 
Model-It and Stella software to provide models of electrical and other systems 
selected for development as an engaged learning unit. Studies indicate positive 
impact on student learning when modeling software has been implemented in 
classroom settings to address solutions to real world problems (Soloway et al, 1997). 



Besides training, staff received the hardware and software necessary to utilize these 
unique technologies in their classrooms and school districts. Videotapes were also 
provided for teachers to refresh their training as needed at their home sites. The 
videotapes included methodologies for: 1) use of Excel software for data collection 
and analyses of budgets, 2) use of the Model-It software to predict system effects of a 
given problem of choice, 3) use of the Stella software helpful in conducting a 
prescribed energy audit and capable of adaptation to other problems and settings, 
and 4) introduction in the creation of web-based home pages, e-mail 
communications, and other internet-based technology uses. 



Technological Resources 

During a planned workshop, program participants were trained by the NCSA staff 
in using NetFusion to create web-based home pages for their home school. This 
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internet technology allowed teachers to open e-mail accounts to communicate with 
other project teams across the state, and with ISBE, NCSA and NCRTEC program 
staff. This software also extended internet capabilities to the classrooms of each of 
the sixth grade teachers for use within the project and in other instructional areas of 
choice. Additional support help and training was provided for project teams at their 
home schools as needed for successful implementation of school-based web pages, 
internet access, and e-mail communication. The varying hardware and related 
software necessary for each participating school to implement this phase of the 
project was provided by grant funds on an as needed basis. 



Student Projects 

Many of the above mentioned project goals, components and activities were 
addressed through development and implementation of an energy project at each of 
the participating schools. The energy project was used as a model unit for teachers 
to emulate since it incorporated the engaged learning, action research and 
performance-based assessment aspects of the project with easily accessible and state 
of the art technology. 

In some cases the energy project did not meet the curricular needs of the particular 
school, so it was altered to align with current school standards (i.e. became an 
environmental project involving recycling of community waste products or ensuring 
the quality of local water). 

A typical project required students to: 

• determine a relevant school research project, 

• study various project parameters in depth, 

• use modeling software to test possible intervention strategies, 

• carry out and monitor strategies through software modeling, 

• prepare a report of findings, and 

• present results to various community audiences. 
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Program Evaluation 

The evaluation portion of the grant assesses the extent to which each of the program 
goals has been met and also indicates corresponding merit and worth of the stated 
goals. Sometimes the evaluation activities were formative in nature, taking place 
during the actual implementation phase of the grant project and providing feedback 
for program improvements. Other evaluation techniques were strictly summative, 
occurring at the end of the project and providing commentary about merit and 
worth of the program (Worthen & Sanders, 1987). 

Ensuring that evaluation activities address both formative and summative aspects of 
the program being studied is an actual strength of the design of the project and the 
evaluation study. 



Confirming Evidence 

The evaluation plan utilized a confirming evidence approach to gauge program 
effectiveness, and provide insight and understanding for continued planning of the 
partnership activities for the duration of the project. For example, each program 
goal and research question addressed in the evaluation plan were studied both 
formatively and summatively, thus providing feedback on current program efforts 
and also provide recommendations for addressing long-term planning of specific 
issues and concerns (Patton, 1990). 

Each program goal was said to have been achieved or met by Confirming Evidence, if 
and when multiple data measures confirmed same. Program goals and research 
questions which produced conflicting or differing pieces of data measures were said 
to have produce Mixed Evidence, and were deemed inconclusive. Disconfirming 
Evidence was the term used when multiple data measures verified that program 
goals have not been successfully met (Rudy, 1999). 



Triangulation 

A method of data collection used by many researchers to ensure accurate findings is 
called triangulation. Basically, triangulation requires that multiple data measures 
(i.e. minimally three distinct measures) be used to produce evidence related to each 
of the questions studied or addressed in the evaluation plan (Miles & Huberman, 
1984; Denzin & Lincoln, 1994). 

This variety and multiplicity of data measures strengthens the validity and 
reliability of any conjectures ultimately made about progress made toward 
achieving the program goals. The evaluation plan integrated triangulation with the 
confirming evidence approach as the base of all evaluation activities. 

Latest findings from the research community support a mix of qualitative and 
quantitative data sources when conducting program evaluation research (Frechtling 
& Sharp, 1997). A mix of qualitative and quantitative data were collected to assess 
the impact of the AHAs project activities upon attainment of program goals. 
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Performance-based Data 

Criterion-referenced testing (CRT) has been popular for many years since it allows 
teachers and school districts to define and articulate a successful level of student 
performance to indicate mastery or attainment of a given outcome or objective. In 
the past decade revised forms of CRT's (i.e. performance-based measures) have 
become accepted standards of practice. 

Performance-based assessment often allows students to demonstrate what they 
know about a particular topic or subject and to show how it can be used or applied. 
This concept of what we want students "to know and do" is at the heart of 
standards-based assessment and has been implemented successfully in many local 
school improvement programs. 

For the AHAs project, several broad categories of performance-based assessment 
measures were utilized. These measures included: teacher artifacts exemplified by 
curriculum planning documents; student artifacts such as student work and student 
led presentations; teacher judgments gathered through group and individual 
feedback techniques, and teacher observations of student performance while 
working with the modeling software during the engaged learning units. 



Evaluation Standards 

* The American Evaluation Association, a professional organization dedicated to 
implementation and advancement of state of the art evaluative techniques in various 
settings, has created Program Evaluation Standards. These benchmarks for 
evaluators are used to ensure that evaluation provides accurate, valid and reliable 
information that meets the needs of the intended stakeholders. It will be the duty of 
the evaluator to ensure that these Standards are fully implemented and utilized as 
appropriate for the purposes of this study. Major themes of the Program Evaluation 
Standards include the utility, feasability, propriety, and accuracy of the evaluative 
activities. 



Documentation & Instrumentation 

Several broad categories of data measures were utilized in data collection and data 
analyses. Again both qualitative (i.e. performance-based) and quantitative measures 
of both teacher and student performance were utilized. Data from each of the above 
categories was reported in this evaluation section as it pertained to one or more of 
the project goals. 

Qualitative measures used to gauge project activities were: 

• teacher artifacts (e.g. engaged learning planning documents; action 
research plans), 

• student artifacts (e.g. samples of student work; student presentations), 

• teacher judgement (e.g. focus-group feedback collected from teachers), 
and 
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• observation (e.g. NCSA staff observing teacher and student use of 
technology). 

Quantitative measures included: 

• student performance on Illinois Standards Achievement Test (IS AT), and 

• local assessment measures used in school improvement planning (SIP). 



Interpretation & Analyses 

Again, triangulation of all program goals and research questions resulted in 
multiple data measures. The confirming evidence approach produced 
documentation verifying one of the following findings for each item of analyses: 
Confirming Evidence; Mixed Evidence; or, Disconfirming Evidence. 

This approach was utilized with all qualitative and quantitative data for it is 
appropriate and provides the information needed to inform the various stakeholders 
(Joint Committee on Standards for Educational Evaluation, 1981). Additionally, if 
and when quantitative data was analyzed, an appropriate and suitable statistic of 
choice was also utilized per accepted guidelines published and accepted by the 
research community. 



Evidence of Claim Statements 

The grant proposal stated 11 distinct claim statements involving student 
performance, teacher performance, or technology utilization. Each of these claim 
statements was reviewed in light of documentation and data gathered through both 
the formative and summative phases of the evaluation. Multiple data sources were 
collected and appropriately analyzed for each of the claim statements. Judgements 
about the claim statements were then made based upon the collected evidence. 

Summary Chart 1 (found on page 16 of this report) lists the 11 intended claim 
statements posited to be achieved by successful completion of the grant activities 
along with a judgement about the related evidence. 

Student Performance Claims 

Claims 1 & 2 relate to student performance. Evidence reviewed for these claims 
included the following documentation: 

• IGAP & IS AT student assessment data, 

• teacher designed performance-based assessments, 

• focus group feedback of teachers, 

• portfolios of student work, and 

• student developed internet web sites. 

Student assessment data of ISAT was not available for review at the time of this 
evaluation (due out November, 1999). For this reason the term mixed evidence was 
used to describe student performance in Claim 1. It could be that ISAT data will 
support this claim in the future. 
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Regarding claim 2, multiple pieces of evidence support the claim that students have 
indeed demonstrated understanding of the Illinois Learning Standards. 

Teacher Performance Claims 

Claims 3, 4, 5 & 6 relate to teacher performance. Evidence reviewed for these claims 
included the following documentation: 

• use of teacher developed engaged learning modules, 

• teacher use of interactive software, 

• construction of teacher designed performance-based assessments, 

• implementation of action research projects, 

• focus group feedback of teachers, 

• establishment of community partnerships, 

• focus group feedback of teachers, 

® portfolios of student work, and 

• student developed internet web sites. 

Regarding claims 3-6, multiple pieces of evidence support the claim that teachers 
have indeed designed curricula and implemented instructional units directly 
relating to the Illinois Learning Standards. Thus, confirming evidence exists to 
support these claims. 

Technology Claims 

Claims 7, 8, 9, 10 & 11 relate to the design and implementation of technology to 
improve both teacher and student performance. Evidence reviewed for these claims 
included the following documentation: 

• direct observation of teacher and student use of technology, 

• teacher designed instructional units, 

• teacher artifacts created through utilization of technology, 

• student products resulting from technology, 

• focus group feedback of teachers, 

• portfolios of student work, and 

• student developed internet web sites. 

Claims 8-11 are fully supported by confirming evidence. Claim 7 has produced 
mixed evidence due to the lack of technology support and various hardware 
limitations inherent to specific school sites and districts. This is not a reflection of 
the project as much as it is the state of equal access to technology by all students 
across the state. 

Confirming evidence is found top support 9 of the 11 claims originally posed by the 
grant, with the 2 remaining claims producing mixed evidence. Please note that no 
claims produced discontinuing evidence, thus concluding that all claims were 
realized to some extent, with the vast majority (9 of 11 = 82%) of the claims being 
fully achieved. 
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Summary Chart 1: Claim Statements 



Claim 


Students will: 


Confirming 

Evidence 


Mixed 

Evidence 


Disconfirming 

Evidence 


i 


achieve high standards in 
mathematics, science and reading. 




V 




2 


demonstrate understanding of 
standards in mathematics, science, 
and reading that had been 
previously inaccessible to them. 









Claim 


Teachers will: 


Confirming 

Evidence 


Mixed 

Evidence 


Disconfirming 

Evidence 


3 


use engaged learning models that 
incorporate appropriate 
technologies. 


v 






4 


develop and use performance- 
based assessments aligned with 
the Illinois Learning Standards. 








5 


plan, carry out and evaluate 
action-based research projects. 


/ 






6 


establish school partnerships 
within the community and 
globally. 









Claim 


Technology will: 


Confirming 

Evidence 


Mixed 

Evidence 


Disconfirming 

Evidence 


7 


be connective, ubiquitous, 
interconnective, and designed for 
equitable use. 








8 


be designed for user contributions 
and collaborative and engaged 
learning projects. 








9 


demonstrate engagibility. 


yf 






10 


be use friendly, fast, available for 
training and support, and have 
the ability to provide immediate 
feedback. 


y/ 






11 


demonstrate functionality through 
diverse tools, media use and 
authoring support designed to 
help students achieve standards. 


yf 
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Evidence of Completion of Objectives 

Summary Chart 2 (found on page 18 of this report) documents the three major goals 
described in the grant proposal and related objectives. Again the extent to which 
goals and objectives were successfully met or achieved was directly linked to the 
evidence provided. 

Regarding Goal 1 and student achievement, all 3 objectives were found to be 
supported by confirming evidence. The evidence included: teacher artifacts, student 
artifacts, teacher judgement, and observation. 

Goal 2 relating to professional development of staff was also fully supported by 
confirming evidence. Each of the 4 objectives met the necessary criteria by: teacher 
artifacts, teacher judgement and observation. 

Goal 3 of the grant involved the establishment of community partnerships by 
teachers. Confirming evidence produced positive verification of Objective 1 related 
to Goal 3, but not for Objective 2. Again the hardware and technological support 
issues at the local sites resulted in conflicting or mixed evidence for Objective 2. 

Evidence of Goal Attainment 

Summary Chart 3 (found on page 19) examines the issue of attainment for stated 
grant goals. Three goals were developed and posed as indicators of success for the 
AHAs project. Confirming evidence exists that all three goals have been fully 
realized as originally written in the grant proposal. 

Teachers have help students achieve high academic standards as evidenced by : 
student artifacts , student performance, teacher artifacts, and teacher judgement. 
Thus, Goal 1 has been realized. 

Goal 2 provided staff development opportunities for teachers evidenced by: 
observation, teacher artifacts, student artifacts, and teacher judgment. Again, 
confirming evidence exists to verify this goal. 

Community partnerships were the focus of Goal 3. Evidence such as, teacher 
artifacts, student artifacts and teacher judgment verify that these partnerships have 
been established. Goal 3 is also deemed to possess confirming evidence. 

It can be said that 100% of the goals have been realized through the implementation 
of the Project AHAs grant activities. Not all objectives were completely realized, nor 
did all original claim statements come to fruition. This is to be expected with the 
size and scope of this grant project. But when taken as a whole, the goals of the 
grant have been verified through use of multiple performance-based assessment 
measures. 
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Summary Chart 2: Grant Objectives 



Goal 1 


Help students achieve high academic 
standards in math, science and reading. 


Confirming 

Evidence 


Mixed 

Evidence 


Discontinuing 

Evidence 


Objective 1 


Identify math, reading and science 
standards that are difficult for students 
to achieve. 


V 






Objective 2 


Carry out 5 full day technology-rich 
interdisciplinary learning workshops for 
6th grade teachers. 


V 






Objective 3 


Implement engaged learning and 
technology-rich projects. 


V 







Goal 2 


Provide staff development 
opportunities for teachers to develop, 
implement and evaluate technology- 
rich reality-based learning projects. 


Confirming 

Evidence 


Mixed 

Evidence 


Discontinuing 

Evidence 


Objective 1 


Plan and implement reality -based, 
technology rich, interdisciplinary units. 


V 






Objective 2 


Help teachers become more 
technologically literate. 


V 






Objective 3 


Teachers will learn how to plan and 
carry out action based research projects. 


V 






Objective 4 


Consortium districts and private schools 
will work collaboratively to plan for 
ongoing and sustained professional 
development. 


V 







Goal 3 


Establish community collaboration 
between participating 6th grade 
students. 


Confirming 

Evidence 


Mixed 

Evidence 


Disconfirming 

Evidence 


Objective 1 


Involve diverse representation in 
community partnerships. 


V 






Objective 2 


Develop an internet-based system for 
districts to share engaged learning 
projects. 




V 
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Summary Chart 3: Project Goals 



Project AHAs Goals 


Confirming 

Evidence 


Mixed 

Evidence 


Discontinuing 

Evidence 


l 


Help students achieve high 
academic standards in math, 
science and reading. 


V 






2 


Provide staff development 
opportunities for teachers to 
develop, implement and evaluate 
technology-rich reality-based 
learning projects. 


V 






3 


Establish community 
collaboration between 
participating 6th grade students. 


V 







Statement of Findings 

Based upon the review of the evidence presented the following claim statements can 
be made regarding the AHAs project: 

Students have : 

• met or exceeded high standards in mathematics, science and reading, and 

• have demonstrated understanding of the related Illinois Learning 
Standards. 

Teachers have : 

• successfully used engaged learning principles in instructing students, 

• developed and implemented performance-based assessment 
measurement procedures, 

• aligned curricula with the Illinois Learning Standards, 

• planned and implemented an action research model, and 

• established collaborative community partnerships for their students. 

Technology has : 

• been implemented equitably among school districts, 

• been designed to foster communication 
within and among school districts, 

• demonstrated engagability by both teachers and students, 

• been introduced through group and individual training, and 

• provided diverse tools and media to support student achievement. 

Some of the teacher artifacts used to verify these findings and claim statements 
were: engaged learning units, performance-based assessments, action research 
models, and locally developed internet web sites. 
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Student artifacts included student portfolios and student presentations conducted 
with PowerPoint software, along with student use of modeling software. 

Technology performance indicators included: improved technological literacy of 
teachers and students, focus group feedback, random observations, e-mail surveys, 
technology training, website development, specific software printouts such as 
graphs and energy models, and student-led presentations to students, faculty and 
boards of education. 

Summary of Findings 

It can also be said that 100% of the schools involved successfully utilized technology 
to improve student performance as measured by various performance-based 
assessment standards. The AHAs program has proven that a technology rich 
classroom can be used to improve student performance in the collection, 
interpretation, and analyses of data. The project also confirms that students of 
varying abilities, disparate socioeconomic levels and learning styles can all benefit 
by integrating classroom teaching and learning with various technologies. 

Next Steps 

Continued work is needed in developing and refining appropriate learning 
technologies to be used by adolescent learners within various subject domains. 
Much more staff development and training is necessary to ensure teacher comfort 
and expertise in utilizing and implementing the latest learning technologies within 
individual schools and communities. Additional hardware and software is 
warranted to ensure equal access by all teachers and all students to the latest 
technological innovations, and to communicate and share information electronically 
with others engaged in the standards-based improvement of schools. 



Future Directions 

The Achieving High Academic Standards (AHAs) Project exemplifies the collective 
wisdom and hard work of a number of individuals and organizations from across 
the state of Illinois. The project was impactful due to the leadership exhibited by the 
Illinois State Board of Education, the National Computational Science Alliance 
(NCSA) and the North Central Regional Technology in Education Center 
(NCRTEC). Thirty-eight classroom teachers from eight school districts and three 
private schools now have the tools to extend the learning of their students. 

The project was successful because it set high standards and put in motion the steps 
necessary to carry out and achieve its goals, objectives and claims. It was designed 
to positively impact the professional growth of teachers, reform curricula, and 
ultimately improve student learning through the consistent and varied use of 
technology. This is just one success story of many that highlights the efforts of 
collaborative partnerships ensuring that students can and do achieve the Illinois 
Learning Standards. 
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Additional Information 



For additional information related to any aspect this project please contact: 

• Bill Conrad, Associate Superintendent Learning Technologies, Illinois State Board 

of Education (wconrad@smtp.isbe.state.il). Principal Investigator Project 
AHAs 

• Dennis Rudy, Indiana University South Bend (drudy@iusb.edu), program 
evaluation and performance assessment 

• Raul Zaritsky, Senior Research Scientist, National Computational Science 

Alliance, (raulz@ncsa.uiuc.edu), modeling software and learning technologies 





Achieving High Academic Standards (AHAs) Project 

r' 

Executive Summary 

August, 1999 , 

Dennis W. Rudy, Ph.D. 

Indiana University South Bend 



Project Overview 

The Illinois State Board of Education has 
partnered with the National Computational 
Science Alliance (NCSA) and tne North 
Central Regional Technology in Education 
Center (NCRTEC) to provide staff 
development training along with 
technological software, hardware and 
support to improve the performance of over 
1000 sixth grade students from across the 
state in the areas of mathematics, science 
and reading. The project involved teachers, 
administrators, and support staff from eight 
public school districts and three private 
schools from thirty-eight different 
classrooms and eight distinct regional 
communities. 

The classrooms selected for the project 
contained students reflecting the attributes 
of low socioeconomic status, high mobility 
rates in transferring both in and out of 
school> and a large percentage of students 
receiving special education services. The 
program participants were provided five 
days of training in the engaged model of 
learning and in the use of specific modeling 
software (i.e. both Stella and Model-It). 



Communication among participants was 
enhanced through the use of e-mail and 
other web-based technologies. 

Teachers were asked to plan, implement 
and evaluate an action research project at 
their school involving students in collecting 
data and information on a real problem of 
choice (see one example below from Cahokia 
School District, Centerville Elementary School). 




This project was funded by an Illinois State Board of Education Technology Literacy Challenge grant. 
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Projects completed included; energy audits 
in which students devised ways to decrease 
the cost and consumption of electrical 
energy; a recycling project designed to 
manage student use of paper and related 
implications for saving trees and reducing 
local landfill waste; and, collaboration with 
a local park district to monitor and improve 
the water quality of a recreational lake; 

Performance assessment measures were 
utilized to gauge the merit and worth of the 
project; Feedback from the program 
participants was utilized to improve future 
efforts at local school sites in improving 
student performance in achieving state and 
local standards related to data collection, 
representation and interpretation. 



Confirming Evidence 



le evaluation plan utilized a confirming 
evidence approach to gauge program 
effectiveness, and provide insight and 
understanding for continued planning of 
the partnership activities for the duration of 
the project. For. example, each program 
goal and research question addressed was 
reviewed and studied both forma tively and 
summatively, thus providing feedback on 
current and past project efforts along with 
recommendations for addressing long-term 
program planning issues (Patton, 1990; 
Worthen & Sanders, 1994). . 

Each program goal was said to have been 
achieved or met by Confirming Evidence, if 
and when multiple data measures 
confirmed same. Program goals and 
research questions which produced 
conflicting or differing pieces of data 
measures were said to have produce Mixed 



Evidence, and were deemed inconclusive. . 
Discon firming] Evidence was . the term used 
when multiple data measures verified that 
program goals have not been successfully, 
met (Rudy, 1999). 



Multiple Assessments 

A method of data collection used by many 
researchers to ensure accurate findings is 
called triangulation. This procedure 
requires that multiple data measures (i.e. 
minimally three distinct measures) be used 
to produce evidence related to each of the 
questions addressed in the evaluation plan 
(Miles & Hu berm an, 1984; Denzin & 
Lincoln,. 1994). This variety and multiplicity 
of data measures strengthens the validity 
and reliability of any conjectures ultimately 
made regarding progress made toward 
achieving the program goals. The 
evaluation plan for this project integrated 
triangulation with the confirming evidence 
approach as the base of ' the evaluation 
activities. Latest findings from the research 
community support a mix of qualitative and 
quantitative data sources when conducting 
program evaluation research (Frechtling & 
Sharp, 1997). 



Performance-based Data 

Criterion-referenced testing (CRT) has been 
popular for many years since it allows 
teachers and school districts to define and 
articulate a successful level of student 
performance to indicate mastery or 
attainment of a given outcome or objective. 
In the past decade revised, forms of CRT’s 
(i.e. performance-based measures) have 
become accepted standards of practice. 

Performance-based assessment often allows 
students to demonstrate what they know 
about a particular topic or subject and to 
show how it can be used or applied. . This 
concept of what we want students "to know 



heart of standards-based 
has been implemented 
local school 



and do" is at the 
assessment and 
successfully in many 
improvement programs. 

For the AH As project, several broad 
categories of performance-based assessment 
measures were utilized. These measures 
included: teacher artifacts exemplified by 



curriculum planning 



artifacts such as student work and student 



documents; student 
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• been introdueed through group and 
individual training, and 

• provided diverse tools and media to 
support student achievement. 



led presentations; teacher judgments 
gathered through group and individual 
feedback techniques, and teacher 
observations of student performance while 
working with the modeling software during 
the engaged learning units. 

Findings & Implications 

The evaluation activities have produced 
confirming evidence for each of the three 
major project goals (see chart below). 

Pro gram Goals 

1) Help students achieve high academic 
standards in math, science and reading; 

2) Provide staff development opportunities for 
teachers to develop,, implement and evaluate 
technology-rich reality-based learning \ 
projects; and 

3) Establish community collaboration among 
participating 6th grade students. 

The following claim statements can be said of 
the AHAs project: 

Students have : 

• met or exceeded high standards in 
mathematics, science and reading, and 

• have demonstrated understanding of 
the related Illinois Learning Standards. 

Teachers have : 

• successfully used engaged learning 
; principles in instructing students, 

• developed and implemented 
performance-based assessment 
measurement procedures, 

• aligned curricula with the Illinois 
Learning Standards, 

• planned and implemented an action 
research model, and 

• established collaborative community 
partnerships for their students. • 

Technology has : 



Some of the teacher artifacts used to verify 
these findings and claim statements were: 
engaged learning units, performance-based 
assessments, action research models, and 
locally developed internet web sites. 

Student artifacts included student portfolios 
and student presentations conducted with 
PowerPoint software, along with student 
use of modeling software. 

Technology performance indicators 
included: improved technological literacy of 
teachers arid students, focus group 
feedback, random observations,: e-mail 
surveys, technology training, website 
development, specific software printouts 
such as graphs and energy models, and 
student-led presentations to students,, 
faculty and boards of education . 




• been implemented equitably among 
school districts, 

• been designed to foster communication 

within and among. school districts, : . 

• demonstrated engagability by both 
. teachers and students, 



Digital photo of teachers engaged in learning to 
use the Stella modeling software taken during 



the November, 
conducted at the 
Illinois. 



1998 two-day. workshop 
NCSA center in Champaign, ] 
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Next Steps 

It can also be said that 100% of the schools 
involved successfully utilized technology to 
improve student performance as measured 
by various performance-based assessment 
standards. The AH As program has proven 
that a technology rich classroom can be 
used to improve student performance in the 
collection, interpretation, and analyses of 
data. The project also confirms, that 
students of varying abilities, disparate 
socioeconomic levels and learning styles 
can all benefit by integrating classroom 
teaching and learning with various 
technologies; - ' 



Continued work is needed in developing 
and refining appropriate learning 
technologies to be used by adolescent 
learners within various subject domains. 
Much more staff development and training 
is necessary to ensure teacher comfort and 
expertise in utilizing and implementing the 
latest learning technologies within 
individual schools and communities. 
Additional hardware and software . is 
warranted to ensure equal access by all 
teachers and all students to the latest 
technological innovations, and to 
communicate and share information 
electronically with others engaged in the 
standards-based improvement of schools. 



Want More Information? 

For additional information related to any 
aspect this project please contact: 

• Bill Conrad, Associate Superintendent 
Learning Technologies, Illinois State Board 
of Education (wconrad@smtp.isbe.state.il). 
Principal Investigator Project AHA s 

• Dennis Rudy, Indiana University South 
Bend (drudy@iusb.edu), program 
evaluation and performance assessment 

• Raul Zaritsky, Senior Research Scientist, 
National Computational Science Alliance, 
(raulz@ncsa.uiuc.edu), modeling software 
and learning technologies 
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